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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent temi adjustment. See 37 CFR 1.704(b). 

Status 

1 )E1 Responsive to connmunication(s) filed on 09 August 2004 . 
2a)^ This action is FINAL. 2b)n This action is non-final. 

3) 0 Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-38 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) ^ Claim(s) 29-34 is/are allowed. 

6) ^ Claim{s) 1-3.22-28 and 35-38 is/are rejected. 

7) ^ Claim(s) 4-21 is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 13 The specification is objected to by the Examiner. 

10)n The drawing{s) filed on is/are: a)n accepted or b)n objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 

Priority under 35 U.S.C. § 119 

12)n Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)n All b)n Some * c)^ None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. , 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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Detailed Action 

1 . The Examiner in charge of this application has changed to Primary Examiner 
Jeffery A. Brier who may be reached at 703-305-4723. 

Response to Amendment 

2. The amendment filed on 8/9/2004 has been entered. 

Response to Argument 

3. Applicants arguments as they may apply to the new grounds of rejection are not 
persuasive because the newly discovered Chen article, Text Extraction in Videos using 
Topographical Features of Characters, teaches determining text overiayed upon video. 
The article also teaches initially determining an overlay then verifying the overlay. 
Similarly the newly discovered Antani article, Robust Extraction of Text in Video, 
teaches detecting static overlays on video by performing a frame to frame comparison 
of the video. The article also teaches initially determining an overlay then performing 
more processing to verify the overlay. 

Specification 

4. The disclosure is objected to because of the following informalities: in paragraph 
0034 at line 1 figure 1 1 is incorrect because a figure i 1 in not present in the drawings. 
Perhaps applicant intended to make reference to figure 10 since in paragraph 0034 at 
line 1 1 makes reference to figure 10. 

Appropriate correction is required. 



Application/Control Number: 09/935,610 Page 3 

Art Unit: 2672 

Claim Rejections - 35 USC §112 

5. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

6. Claims 28 is rejected under 35 U.S.C. 112, second paragraph, as being indefinite 
for failing to particularly point out and distinctly claim the subject matter which applicant 
regards as the invention. It is not clear if the claimed phrase 'a computer-readable 
medium containing software embodying the method of claim 4" causes the computer to 
implement the method of detecting a pre-existing overlay. If it does not then the claim 
would only be storing a program rather than implementing the method of claim 4. It 
would in essence be storing descriptive data. See MPEP 2106. Applicant could amend 
this claim to claim "a computer-readable medium storing computer executable code for 
causing the computer to implement the method of claim 4" or use similar language. 

Claim Rejections - 35 USC § 102 

7. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for a patent. 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

8. Claims 1-3, 22. 23, 26, and 27 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Byung Tae Chen. Younglae Bae, Tai-Yun Kim, Text Extraction in Videos 
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using Topographical Features of Characters, August 22-25, 1999. IEEE, vol. 2, pages 
1126-1130. 

This article teaches extracting text from video by two main steps of extracting 
candidate areas using topographical features and then verifying text is in those areas. 
Section 3.1 discusses extracting candidate area for text area and section 3.2 discusses 
verification of candidates of text area. 

A detailed analysis of the claims follows. 
Claim 1: 

Chen teaches a method of video processing (see introduction . ) 
comprising: 

extracting a pre-existing overlay present in a video sequence (See the 
introduction, paragraph 1 which discusses text appearing in 
video such as news where is often used to identify people, see 
figure 2, and to place identifying marks, see the upper left and 
lower right corners of figure 2 .) said extracting comprising: 

detecting at least one potential overlay in the video sequence {Section 3 . i 
discusses extracting candidate area for text area,)', and 

verifying that the at least one potential overlay is at least one actual overlay 
{section 3.2 discusses verification of candidates of text area,). 

Claim 2: 

Chen teaches the method of claim 1 , further comprising the step of post-processing at 
least one actual overlay to remove extraneous pixels {Figure i shows the post 
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processing step of removing noise. Noise comprises extraneous 
pixels such as non- character regions inside the character 
regions, see section 3.3, thus, Chen teaches removing extraneous 
pixels.). 

Claim 3: 

Chen teaches the method of claim 2. wherein said step of post-processing comprises 
the steps of: 

computing a variance for each pixel of the at least one actual overlay (section 3 . 3 
discusses removing noise by using Isodata color clustering. The 
verified actual overlay area is analyzed to determine the color 
of each pixel to cluster the pixels in the overlay area.)] and 
comparing the variance with a threshold to determine whether or not the pixel should be 
removed as an extraneous pixel (The size of the color clusters are 
compared and if they are too small the cluster is removed which 
removes the pixels forming each cluster.). 

Claim 22: 

Chen teaches the method of Claim 1 , wherein said step of detecting comprises the step 
of: 

performing template matching to determine the presence of a potential overlay 

{Section 2 and 3.1 discusses using the topological features of 
characters to determine the presence of a potential overlay. 
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Topological features of characters define a template for each 
character or groups of characters .). 

Claim 23: 

Chen teaches the method of claim 22, wherein said step of detecting further comprises 
the step of: 

determining a template (The paragraph before section 3 discusses 
determine n and alpha. The values of n and alpha form a 
template.) to be used in said step of performing template matching. 

Claim 26: 

Chen teaches a computer readable medium containing software embodying the method 
of claim 1 (Chen discusses using a computer to perform the text 
extraction in section 4. The discussed Pentium 4 computer using 
a program written in Visual C++ Ver* 5.0 has the program stored 
in a computer readable medium, the disk drive and RAM.), 

Claim 27: 

Chen teaches a computer system comprising: 

a computer (Chen discusses using a Pentium 4 computer to perform 
the text extraction in section 4.); and 
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a computer readable medium containing software embodying the method of claim 1 

{Chen discusses using a computer to perform the text extraction 
in section 4. The discussed Pentium 4 computer using a program 
written in Visual C++ Ver. 5.0 has the program stored in a 
computer readable medium, the disk drive and RAM). 



9. Claims 1. 22, 23, 24, 25, and 35-38 are rejected under 35 U.S.C. 102(a) as being 
anticipated by S. Antani. D. Crandall, R. Kasturi, Robust Extraction of Text in Video, 
Sept 3-7. 2000. IEEE. vol. 1, pages 931-834. 

This article teaches detecting static overlays on video by performing a frame to 
frame comparison of the video. In the section 3, second paragraph at lines 7-11 
"artificial caption text" and "scene text occurring naturally in a video frame" is discussed. 

A detailed analysis of the claims follows. 
Claim 1: 

Antani teaches a method of video processing (See^ introduction,) 
comprising: 

extracting a pre-existing overlay present in a video sequence (See the 
introduction, paragraph 1 second column which discusses text 
appearing in video,) said extracting comprising: 

detecting at least one potential overlay in the video sequence {Section 2 
discusses three stages, the detection, localization, and 
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segmentation stages. The detection stage detects a potential 
overlay .y, and 

verifying that the at least one potential overlay is at least one actual overlay 

{Section 2 discusses the localization stage which uses many 
methods to localize the text. Section 2 discusses using many 
different localization algorithms whose outputs are fused in the 
spatio-temporal decision fusion module over multiple frames to 
verify that a potential text is text. Section 2 also discusses 
using a tracking stage, this would inherently verify the 
potential text is an actual text.). 

Claim 22: 

Antani teaches the method of Claim 1 , wherein said step of detecting comprises the 
step of: 

performing template matching to determine the presence of a potential overlay 

{Section 2 discusses the detection of potential overlay in the 
detection stage which consists of many different localization 
algorithms whose outputs are fused in the spatio-temporal 
decision fusion module over multiple frames. In order to 
determine if text exists then predefined knowledge of the text 
is compared with the current image to determine if a match 
exists. Predefined knowledge of the text is a template.). 
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Claim 23: 

Antani teaches the method of claim 22. wherein said step of detecting further comprises 
the step of: 

determining a template to be used in said step of perfomning template matching 

(inherently at some time the templates used by the program were 
determined.). 

Claim 24: 

Antani teaches the method of claim 22, wherein said step of verifying comprises the 
steps of: 

performing frame-to-frame correlation of said potential overlay {Section 2 
discusses using many different localization algorithms whose 
outputs are fused in the the spatio-temporal decision fusion 
module over multiple frames,)] and 

comparing a result of the frame-to-frame correlation with a threshold to determine 
if the potential overlay is an actual overlay or not {in order to determine if 
text exists then predefined knowledge of the text is compared 
with the current image to determine if a match exists. 
Predefined knowledge of the text is a template of thresholds.). 
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Claim 25: 

Antani teaches the method of claim 24, wherein said step of performing frame-to-frame 
correlation (See the discussion above for claim 24 .) comprises the steps of: 

forming a mean square error over a set of frames from said video sequence, 
averaged over all of the pixels in said potential overlay {This claim does not 
claim a use for the mean square error, thus, a reference that 
forms a mean square error over a set of frames teaches the 
claim. This claim does not claim how the mean square error is 
formed, thus, a reference that inherently foirms the error 
teaches the claim. The specification in paragraph 0039 sets 
forth a specific formula for calculating the mean square error, 
however, the claim only broadly claims how the claimed mean 
square error is calculated. Although the claims are interpreted 
in light of the specification, limitations from the 
specif ication are not read into the claims. See In re Van 
Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). The 
disclosed formula determines the average difference in 
intensities between a current frame and a subsequent frame. 
Antani inherently forms the mean square error since Antani in 
the localization stage fuses over several frames decisions from 
many localization algorithms which inherently has determined the 
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average difference in intensities between frames in order to 
determine if text exists.). 

Claim 35: 

Antani teaches a method of processing video, comprising: 

extracting a pre-existing graphical (Jn the section 3 second paragraph lines 
7-11 ''artificial caption text'' and "scene text occurring 
naturally in a video frame'' is discussed. Artificial caption 
text is graphical because graphical includes many objects 
including text . ) overlay present in a video sequence, said extracting comprising: 
detecting at least one potential overlay in a the video sequence (section 2 
discusses three stages, the detection, localization, and 
segmentation stages. The detection stage detects a potential 
overlay. ), said detecting comprising the step of: 

performing template matching (Section 2 discusses the detection of 
potential overlay in the detection stage which consists of many 
different localization algorithms whose outputs are fused in the 
spatio-temporal decision fusion module over multiple frames. In 
order to determine if text exists then predefined knowledge of 
the text is compared with the current image to determine if a 
match exists. Predefined knowledge of the text is a template.)] 
and 
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verifying that the at least one potential overlay is at least one actual overlay {Section 
2 discusses the localization stage which uses many methods to 
localize the text. Section 2 discusses using many different 
localization algorithms whose outputs are fused in the the 
spatio-temporal decision fusion module over multiple frames to 
verify that a potential text is text. Section 2 also discusses 
using a tracking stage, this would inherently verify the 
potential text is an actual text .). said verifying comprising the step of: 
performing frame-to-frame correlation of a potential overlay determined by said 
detecting step (Section 2 discusses using many different localization 
algorithms whose outputs are fused in the spatio-temporal 
decision fusion module over multiple frames.). 

Claim 36: 

Antani teaches the method of Claim 35. wherein said step of detecting further 
comprises the step of: 

determining a template to be used in said step of performing template matching 

{inherently at some time the templates used by the program were 
deteirmined.). 
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Claim 37: 

Antani teaches the method of Claim 36, wherein said step of determining a template 
comprises the step of: 

performing addition or frame-by-frame subtraction of video frames {This claim does 
not define the specifics of the addition of video frames or the 
frame -by -frame subtraction of video frames. This claim does not 
state if pixel values are added or frame numbers are added or if 
as in Antani the results of many frame analyses are fused or 
added or subtracted.). This step does not state what function the addition or 
subtraction performs, thus, the scope of the claim is broad and is met by Antani when a 
template for detection stage is determined since the claim does not claim when the 
template is determined and when the addition or subtraction is performed. Therefore in 
this comprising claim all that is needed is for the reference to teach the claimed steps. 

Claim 38: 

Antani teaches the method of Claim 36. wherein said step of determining a template 
comprises the steps of: 

segmenting video frames into foreground and background objects (Text is 
foreground and video is the background. See the Abstract at the 
next to last and last sentences. Section 1 second paragraph 
lines 8-9. )\ 
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performing correlation tracking to detemnine if any foreground object remains in the 
same absolute location in each video frame {Section 2 discusses using many 
different localization algorithms whose outputs are fused in the 
spatio-temporal decision fusion module over multiple frames to 
verify that a potential text is text. In the last sentence of 
section 2 the article teaches due to the fact that text lasts 
over several frames the text may be determined. The Abstract at 
the last sentence teaches determining if the text is static ,) . 
This step does not state what function the segmenting and correlation tracking 
performs, thus, the scope of the claim is broad and the claim does not claim when the 
template is determined and when the segmenting and correlation tracking is performed. 
Therefore in this comprising claim all that is needed is for the reference to teach the 
claimed steps. 



Prior Art 

10. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Byung Tae Chen, Younglae Bae, Tai-Yun Kim, Automatic Text Extraction in 
Digital Videos using FFT and Neural Network. August 22-25, 1999, IEEE, vol. 2. pages 
1112-1115. This article teaches extracting text from video by two main steps of 
extracting candidate areas using FFT characteristics of text and a neural network and 
then verifying text is in those areas. Section 3.1 discusses extracting candidate area for 



Application/Control Number: 09/935,610 
Art Unit: 2672 

text area by using the neural network and 
verification of candidates of text area. 



Page 15 

to extract text and section 3.2 discusses 



Allowable Subject Matter 

1 1 . Claims 4-21 are objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the 
base claim and any intervening claims. Claim 28 would be allowable if rewritten to 
overcome the rejection(s) under 35 U.S.C. 112, 2nd paragraph, set forth in this Office 
action and to include all of the limitations of the base claim and any intervening claims. 

1 2. Claims 29-34 are allowed. 

13. The following is a statement of reasons for the indication of allowable subject 
matter: 

Claims 4-21 and 28-34: 

The prior of record fails to teach or suggest detecting the potential overlay by 
using wavelet decomposition on the video sequence, extracting features based on the 
wavelet decomposition, and performing neural network processing on the extracted 
features. 
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14. Applicanfs amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 

§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

1 5. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffery A Brier whose telephone number is 703-305- 
4723 until the move and after the move the number will be 703-571-7656. The 
examiner can normally be reached on M-F from 6:30 to 3:00. If attempts to reach the 
examiner by telephone are unsuccessful, the examiner's supervisor, Michael Razavi, 
can be reached at (703) 305-4713). The fax phone number for the organization where 
this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
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published applications may be obtained from either Private PAIR or Public PAIR. 
Status infonnation for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 




Jeffery A Brier 
Primary Examiner 
Art Unit 2672 



